Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial e2e tests #136

Merged
merged 34 commits into from
Jul 5, 2023
Merged

Initial e2e tests #136

merged 34 commits into from
Jul 5, 2023

Conversation

astefanutti
Copy link
Contributor

@astefanutti astefanutti commented Jun 13, 2023 •
edited
Loading

This PR adds e2e tests for the CodeFlare stack, more specifically, MCAD, KubeRay/Ray and the CodeFlare SDK.

It includes:

  • A e2e test suite with 3 initial tests:
    • MNIST training submitted as a RayJob to a RayCluster managed by MCAD
    • MNIST training submitted as a batch Job managed by MCAD
    • MNIST training on a Ray cluster via the CodeFlare SDK
      (only runs on OpenShift, requires K8s Support for SDK codeflare-sdk#146)
  • A GitHub Actions workflow, that runs e2e tests against a KinD test cluster.
  • The parameterisation of the components under tests, so the test suite can be reused by downstream projects.
  • The documentation on how to run / debug the e2e tests locally.

The execution of the e2e workflow currently takes around 14 minutes on average. A third of that execution time corresponds to the build of the operator container image, which may be optimised in the future, by caching layers for example.

Fixes #4.

@openshift-ci
Copy link

openshift-ci bot commented Jun 13, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@astefanutti astefanutti mentioned this pull request Jun 14, 2023
Merged
jiripetrlik
jiripetrlik reviewed Jun 16, 2023
View reviewed changes
.github/workflows/e2e_tests.yaml Outdated Show resolved Hide resolved
.github/workflows/e2e_tests.yaml Show resolved Hide resolved
.github/workflows/e2e_tests.yaml Outdated Show resolved Hide resolved
@jiripetrlik jiripetrlik mentioned this pull request Jun 19, 2023
Closed
@astefanutti astefanutti marked this pull request as ready for review June 27, 2023 15:53
sutaakar
sutaakar reviewed Jun 28, 2023
View reviewed changes
README.md Show resolved Hide resolved
sutaakar
sutaakar reviewed Jun 28, 2023
View reviewed changes
README.md Outdated Show resolved Hide resolved
@sutaakar
Copy link
Contributor

@astefanutti Would it have sense to add comments to the tests describing what functionality is tested in specific tests? Something like a description from test strategy ADR.

@astefanutti
Copy link
Contributor Author

@astefanutti Would it have sense to add comments to the tests describing what functionality is tested in specific tests? Something like a description from test strategy ADR.

@sutaakar yes, absolutely. We can start with commenting the test case functions, and research ways to structure more that information in the next cycles.

KPostOffice
KPostOffice reviewed Jun 28, 2023
View reviewed changes
Copy link
Collaborator

@KPostOffice KPostOffice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions. I'm unfamiliar with golang testing.

.github/workflows/e2e_tests.yaml Show resolved Hide resolved
test/support/gomega.go Outdated Show resolved Hide resolved
.pre-commit-config.yaml Show resolved Hide resolved
test/support/utils.go Show resolved Hide resolved
test/support/client.go Show resolved Hide resolved
@KPostOffice KPostOffice mentioned this pull request Jun 29, 2023
Open
@astefanutti
Copy link
Contributor Author

@astefanutti Would it have sense to add comments to the tests describing what functionality is tested in specific tests? Something like a description from test strategy ADR.

@sutaakar yes, absolutely. We can start with commenting the test case functions, and research ways to structure more that information in the next cycles.

Done with 2045c7a.

@tedhtchang tedhtchang mentioned this pull request Jun 30, 2023
Open
sutaakar
sutaakar approved these changes Jul 3, 2023
View reviewed changes
Copy link
Contributor

@sutaakar sutaakar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@openshift-ci openshift-ci bot added the lgtm label Jul 3, 2023
@sutaakar sutaakar mentioned this pull request Jul 4, 2023
Merged
@openshift-ci openshift-ci bot removed the lgtm label Jul 5, 2023
@openshift-ci openshift-ci bot added the lgtm label Jul 5, 2023
@openshift-ci
Copy link

openshift-ci bot commented Jul 5, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anishasthana, sutaakar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved label Jul 5, 2023
@openshift-merge-robot openshift-merge-robot merged commit 8c1831a into project-codeflare:main Jul 5, 2023
KPostOffice
KPostOffice reviewed Jul 5, 2023
View reviewed changes
README.md Show resolved Hide resolved
@astefanutti astefanutti deleted the pr-e2e branch July 5, 2023 14:23
@ChughShilpa ChughShilpa mentioned this pull request Jul 7, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KPostOffice KPostOffice KPostOffice left review comments

@jiripetrlik jiripetrlik jiripetrlik left review comments

@anishasthana anishasthana anishasthana approved these changes

@sutaakar sutaakar sutaakar approved these changes

@jbusche jbusche Awaiting requested review from jbusche

Assignees

@sutaakar sutaakar

@anishasthana anishasthana

Labels
approved lgtm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Set up e2e tests for codeflare stack
6 participants
@astefanutti @sutaakar @anishasthana @jiripetrlik @KPostOffice @openshift-merge-robot